Skip to content

fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404) - #292

Merged
Dylan-M merged 1 commit into
mainfrom
dylanmyers/pipe-1404-output-latency-ms-buckets
Sep 10, 2026
Merged

Dylan-M merged 1 commit into
mainfrom
dylanmyers/pipe-1404-output-latency-ms-buckets

Conversation

@Dylan-M

@Dylan-M Dylan-M commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Proposed Change

blitz.output.request_latency and blitz.output.hec.ack_poll_latency were declared in seconds and recorded via .Seconds(). Both use OTel's default histogram buckets (0, 5, 10, 25, ... 10000), where the first boundary is 5. Output latencies almost always run well under 5 seconds, so every sample landed in the first bucket. The histograms carried no distribution signal.

Both now record in milliseconds, where the default buckets fit. A new output.DurationMillis helper converts a duration to fractional milliseconds. It stays float so a sub-millisecond sample is not truncated to zero. Truncating would also understate the histogram sum. The request-latency feeders (tcp, otlp_grpc, hec) and the HEC ACK-poll site record through it. Both metric.yaml units change from s to ms.

Stacked on #291.

How the reviewer can validate

Note: the manual validation procedure below is an AI-generated script. Review it before running.

Manual:

  1. make build
  2. Start a throwaway TCP sink: nc -lk 127.0.0.1 5140 >/dev/null &
  3. Run blitz to it with the self-metrics endpoint on:
    ./blitz --generator-type apache-common --generator-apache-common-rate 50ms \
      --output-type tcp --output-tcp-host 127.0.0.1 --output-tcp-port 5140 \
      --metrics-port 9100
    
  4. curl -s localhost:9100/metrics | grep request_latency
  5. Confirm the histogram renders in milliseconds: blitz_blitz_output_request_latency_milliseconds_* (previously ..._seconds_*). On a localhost sink, sub-5ms sends correctly sit in the first bucket. Any latency above 5ms now lands in a higher bucket, where in seconds it would have collapsed into the first.

The HEC ACK-poll histogram (blitz_blitz_output_hec_ack_poll_latency_milliseconds_*) validates the same way against a HEC endpoint with ack enabled.

Automated: go test ./output/.... TestDurationMillis covers the conversion including sub-millisecond preservation. Manual-reader tests assert the ms unit on both blitz.output.request_latency and blitz.output.hec.ack_poll_latency.

Checklist
  • Changes are tested
  • CI has passed

@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from b24bf34 to 792856e Compare August 14, 2026 19:22
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from be2b515 to c52f955 Compare August 14, 2026 19:22
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from c52f955 to e49f88b Compare August 26, 2026 15:43
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 792856e to 91ba669 Compare August 26, 2026 15:43
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 91ba669 to fbc35ad Compare September 4, 2026 13:06
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from e49f88b to 7594cdb Compare September 4, 2026 13:06
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from 7594cdb to a67296a Compare September 4, 2026 21:00
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from fbc35ad to 70e85fe Compare September 4, 2026 21:00
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from a67296a to ba081e7 Compare September 4, 2026 22:54
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 70e85fe to 113a78a Compare September 4, 2026 22:54
@graphite-app
graphite-app Bot changed the base branch from dylanmyers/pipe-1066-lifecycle-metrics to graphite-base/292 September 10, 2026 13:47
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 113a78a to abc82a6 Compare September 10, 2026 13:58
@graphite-app
graphite-app Bot changed the base branch from graphite-base/292 to main September 10, 2026 13:58
@graphite-app

graphite-app Bot commented Sep 10, 2026

Copy link
Copy Markdown

Merge activity

  • Sep 10, 1:58 PM UTC: Graphite rebased this pull request, because this pull request is set to merge when ready.

@Dylan-M
Dylan-M added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit f73965f Sep 10, 2026
18 of 30 checks passed
@Dylan-M
Dylan-M deleted the dylanmyers/pipe-1404-output-latency-ms-buckets branch September 10, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants